Balancing Cost and Quality: An Exploration of Human-in-the-Loop Frameworks for Automated Short Answer Scoring

Short answer scoring (SAS) is the task of grading short text written by a learner. In recent years, deep-learning-based approaches have substantially improved performance SAS models, but how to guarantee high-quality predictions still remains critical issue when applying such models education field. Towards guaranteeing predictions, we present first study exploring use human-in-the-loop framework for minimizing cost while quality allowing model share with human grader. Specifically, introducing confidence estimation method indicating reliability one can utilizing only high results and casting low graders. our experiments, investigate feasibility proposed using multiple methods datasets. We find that allows automatic graders achieve target quality.